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ABSTRACT 



Continuing education students bring with them expectations 
from previous teaching that challenge the effectiveness of usual teacher 
evaluation procedures. This paper presents a validated approach to the 
evaluation of teaching that identifies potential problems with individuals 
and minority groups of continuing students. This approach can be used during 

the course to steer teaching back on course before problems arise. The 

approach, the Three Ability Framework (3AF) , results in a single quantitative 
measure of the quality of teaching that can be used as an administrative 
decision point. The 3AF feedback form takes only a few minutes to complete. 
The teacher explains to the students the three abilities research has 
identified as essential to effective teaching: (1) technical skills; (2) 

professional competence; and (3) professional attitudes. Students are asked 
to rate their assessments of each of these areas and their expectations of 

how the areas should be fulfilled in the course, and instructors rate 

themselves in each area. Student and instructor ratings are compared (student 
change rating is subtracted from lecturer change rating) , and the resulting 
number is makes an evaluative decision point. "Zero" would be a perfect 
indicator of the alignment between student and instructor expectations that 
is indicative of effective teaching. The method allows frequent assessments 
over the duration of the course, permitting adjustments when necessary, and 
it is particularly suited to the evaluation of courses in continuing 
education. (Contains 35 references.) (SLD) 
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Introduction 

( . 

Continuing education students bring with them particular expectations from previous teaching that 
challenge the effectiveness of our usual evaluation procedures. This is especially so for multi-ethnic courses 
where students have a wide variety of cultural expectations of quality teaching. Ubiquitous multi-stakeholder 
cafeteria opinionnaires ask students to endorse values that may be irrelevant, inappropriate or even offensive 
to some students. Some faculty see these course evaluations as little more than popularity contests that 
erode academic standards. These end-of-course post-mortems are of little value for teaching a future 
course with different students and they have no benefit to the students who completed them. This session 
offers participants a validated approach to the evaluation of teaching that identifies potential problems 
with individuals and minority groups of continuing students; an approach that can be used during the 
course to steer teaching back on course before problems arise. The approach results in a single quantitative 
measure of the quality of teaching that can be used as an administrative decision point. This is an integrated 
solution called the 3AF. It is a management framework for promoting a positive teaching, learning and 
research culture in institutions that are responsible for continuing education courses. The 3AF utilises the 
proven influence of assessment in quality assurance to enhance the academic leadership role of the institution 
while respecting our traditional principles of academic freedom and professional responsibility. 

The values emanating from how assessment of students and assessment of faculty are 
integrated into curriculum, research and resourcing are crucial to successful quality assurance 
for continuing education providers. However, often the assessment of individuals and the quality 
assurance review process are, in practice, compartmentalised. Many institutions, now assesses 
the quality of teaching by using anonymous student evaluations solicited towards the end of a 
course. The results of these post-mortem assessments cannot feedback for the benefit of the 
students and the course to which they apply precisely because they are post-mortem evaluations. 
However, they are used by institutions in different contexts, e.g. for promotion and tenure 
decisions. This separation of context and purpose results in lecturers employing techniques of 
managing their teaching that deleteriously skew the institutional learning culture. For example, 
one technique of ensuring high student evaluations is to oveigrade continuous in-course assessment 
and block-teach model examination answers. Students who have adopted a successful learning 
style for such courses are over-confident in their ability to transfer their learning to other courses 
and life situations. They also have diminished self-directed learning skills. The ripple-effect on 
the institutional learning culture is that lecturers on dependent courses must employ similar 
techniques. Dumbing down courses to increase popularity ratings results in many problems for 
the institutional learning culture. One resulting problem for quality assurance is the difficulty 
these students then have in completing research theses and assignments. Research courses and 
research course components, which of necessity are dependent on transfer of learning, critical 
and evaluative thinking and self-directed learning, are being replaced by taught courses. 

Bastick, T. (1999, Oct). Quality assessment for teachers of continuing education and training programs. A 
paper presented at the 1ACET 1999 Conference “Leadership, Creativity and Initiative in Continuing 
Education and Training” for the International Association for Continuing Education and Training, Fort 
Lauderdale, Florida, USA. 
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This paper presents an alternative means of monitoring teaching quality that can be used by 
lecturers and administrators. It is designed to promote a positive teaching and learning culture 
within the institution. This alternative method of assessment was derived from in-depth faculty 
and student interviews. Its criterion of quality has been validated on empirical data by computer 
sensitivity analysis and the assessment process successfully tested in clinical teaching trials. 

This assessment method explicitly promotes students’ understanding and professional 
attitudes, as well as their traditional skills, by operationally defining the assessment of skills, 
understanding and attitudes. It respects professional freedom and the inherent culture of each 
subject area by giving lecturers the responsibility of promoting the culture of their subjects through 
their teaching and assessment of its skills, understandings and values. It allows faculty to adjust 
their in-course teaching and assessment schemes to the changing needs of the student body and 
minority students in their classes. The method results in a single administrative decision-point 
number that is a measure of quality teaching as it applies to (i) a particular individual, or (ii) a 
minority group or (iii) the whole class. 

The 3AF: an eight point overview 

i The measure of quality teaching by student opinionnaires is a problem because this is a core 
negative influence area for the teaching-learning culture of the institution. 

ii The 3AF alternative is a new concept - an alignment method 

iii It aligns and measures staff and student expectations during the course 

iv This alignment predicts course enjoyment and attainment at the end of the course 

v Alignment is on 3 deep structure abilities that underlie quality teaching and learning 

vi Faculty decide the emphasis on each ability that is appropriate for their course 

vii Institutional course/staff development supports faculty in incorporating their chosen emphasis 
into their teaching and course assignments 

viii The measure of staff/student alignment is the measure of teaching quality 

Negative influences on an institution’s teaching and learning culture 

Many organisational factors and resources affect the teaching and learning culture of an educational 
institution. There is a business maxim that states “what gets measured gets done and what gets rewarded 
gets repeated” (Friend, 1 972). In educational institutions what gets rewarded is (i) the assessment of faculty 
teaching and (ii) the assessment of student learning. It is how the assessment of how teaching is done and 
how the assessment of learning is done that has a great influence on the learning and teaching culture of the 
institution (Beichner, 1994; Hake, 1998; Halloun & Hestenes, 1985; Halloun, Hake, Mosca, & Hestenes, 
1997; Hestenes, Wells, & Swackhamer 1992; Hestenes & Wells, 1992; Sokoloff & Thornton, 1997). 
These three aspects (i) teaching (ii) learning and (iii) the assessment of both, are so inter-related that they 
should be considered together. 
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Problems with current assessments 

Unfortunately traditional forms of assessment lack integration, and problems caused by these traditional 
forms of assessment negatively influence the learning and teaching culture of institutions. The common 
traditional assessment of faculty teaching is via the use of student opinionnaires. For example, Seldin 
(1984) found that administrators utilized student opinionnaires in two-thirds of 616 institutions surveyed. 
Student opinionnaires have been criticised as popularity contests, where staff reduce the standards of their 
courses and lead students to expect high grades in order to ‘win their vote’ (Greenwald, 1997; Greenwald 
& Gillmore, 1997; Howard & Maxwell, 1982; Marsh & Dunkin, 1997; Marsh & Roche, 1997, 1998). 
Faculty are expected to change their teaching in view of this anonymous feedback (Moses, 1996). Often 
these opinionnaires are anonymous because students fear retribution for ‘failing’ their teachers or might be 
thought to be soliciting favourable grades for favourable assessments of their teachers. Faculty feel 
uncomfortable in this one-sided contest where they can be anonymously failed by the students they teach, 
yet have little influence over the selection of the students they must teach. As Linda Mabry says on the 
evaluation of teaching “Teachers have little or no control over which students will be assigned to their 
classrooms, how strong or poor their motivation and preparation to learn, their level of transience, or any 
of a number of other pertinent factors ...” (1999, p. 2). 

Some faculty feel that having to change their course emphasis in order to please naive students is an 
infringement of their professional freedom to teach how they believe, in their best professional judgement, 
their subject should be taught (Crumbley, 1 995). It is questionable how much a university should take the 
role of academic leadership or be managed as a business subservient to the student customer where ‘the 
customer is always right’. These influences have been contributing to a negative culture of low academic 
standards, demotivated professionalism and mutual distrust for some years (Arreola, 1983; Cashin, 1983; 
Cherry, Grant, & Kalinos, 1 988). For example, “What is called development, growth, and self-improvement 
today becomes the means by which decisions for institutional personnel management purposes are made 
tomorrow. Faculties become wary and suspicious of this double message involved in the evaluation system” 
(Mark, 1982, p. 168). 

The traditional assessment of student learning is via examinations and coursework assignments of 
various kinds. Here the one-sided game is against the student who has little educational recourse and so 
can only resort to complaint about the course and the faculty - even to the extent of litigation. To protect 
themselves from the ‘court case student’ faculty favour objective assessment that does not expose their 
professional judgements about the students’ work. Such objective assessments tend to emphasise Bloom’s 
lowest cognitive level of rote learning. For example, “McKeachie ( 1 987) has recently reminded educational 
researchers and practitioners that the achievement tests assessing student learning in the sorts of studies 
reviewed here typically measure lower-level educational objectives such as memory of facts and definitions 
rather than higher-level outcomes such as critical thinking and problem-solving that are usually taken as 
important in higher education.” Feldman (1989). Continuing Education students in professional subjects 
rightly devalue courses that emphasise only rote learning, to the extent that they are prepared to cheat in 
order to maximise their meaningless marks. Problem attendance is a feature of such courses - students get 
the lecture notes from those who do attend. Students learn not to criticise the views of faculty but to 
unquestionable do as they are told, and to parrot what they believe Faculty expects them to regurgitate in 
examinations. Such are the negative effects traditional assessments of teaching and learning have on the 
teaching and learning culture of the institution. Research has indicated that these problems are in part due 
to misunderstanding of mutual expectations (Bastick, 1995). Faculty lack clarity in explicitly stating their 
expectations and relating these to their teaching and assessment of the students. The students misunderstand 
what is expected of them and are confused. 
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Positive influences on an institution’s teaching and learning culture 

What is a positive teaching and learning institutional culture? The literature indicates that is one that 
encourages staff and students to be independent critical thinkers developing the attitudes and values of 
their profession. One where students and faculty value and enjoy the work they are doing. One where 
faculty and students respectfully work together based on a foundation of mutual trust. However, if an 
alternative method of assessment is to promote these changes, then first, that method must accommodate 
the wide institutional variations that exist in assessment preferences and aim to improve teaching and 
learning by allowing those lecturers who use it to appreciate more fitting styles of teaching and enable 
them to allow their students to adopt more fitting styles of learning. Secondly, an alternative method of 
assessing teaching and learning must resolve misunderstandings and confusions about mutual expectations 
in order to avoid the problems that lead to a negative teaching and learning culture. Thirdly, an alternative 
method must promote a positive teaching and learning culture by (i) ensuring students and faculty understand 
each other’s expectations and (ii) by ensuring that students and faculty are both working towards the same 
expectations (Abrami, d’Apollonia, & Cohen, 1990; Abrami, 1989; Bastick, 1995; Bortz, 1984; 
L’Hommedieu, Menges, & Brinko, 1990; Miller, 1986; Scriven, 1994, 1995). 

It is also important to separate evaluations of attainment from evaluations of enjoyment, so that 
student evaluations of course quality are not simply ‘smile sheets’ misused as assessments of academic 
attainment (Hake, in press). Hence, the two separated and distinct criteria of effective teaching used by this 
alternate method of assessment are to maximise (i) the academic attainment of the students and (ii) the 
students’ and the lecturer’s enjoyment of the course. The measurable indicator of effective teaching used 
is that the students and the lecturer are working towards the same expectations of the three abilities that 
underlie quality teaching and learning. The construct validity that this measurable indicator assesses the 
criteria is p<0.01 for both (i) and (ii) (Bastick, 1995) 

Interviews with faculty on professional courses have indicated that their implicit expectations can be 
described and assessed in terms of these three abilities (i) technical skills - rote learning, assessed by the 
accuracy of reproduction (ii) professional competence - appropriate transfer of skills to a novel situation, 
assessed by the justification of appropriateness and (iii) professional attitudes - the integration of one’s life 
and work by one’s values and beliefs, assessed by demonstration (Bastick, 1 995). Faculty can be assisted in 
making these expectations explicit and in designing coursework and examinations that offer opportunities 
for assessing these three abilities. This professional development can be expected to improve the quality of 
their teaching (Askew, Brown, Rhodes, Wiliam, & Johnson, 1997). It should be their professional prerogative 
to decide, and their professional responsibility to justify to their peers and their students, the emphasis they 
judge should be given to each of the three abilities on their courses. These judgements will depend on the 
subject, its level and the professional inclination of the lecturer. For example, lecturers on three-year B.Ed 
courses expect an emphasis on technical skills in the first year, moving to an emphasis on professional 
competence in the second year and a greater emphasis on professional attitudes in the third year. 

The Three Ability Framework (3AF) 

Three Ability Framework (3AF) is a complete alternative to post-mortem student opinionnaires and 
their attendant problems. The design of its management framework incorporates the positive influences 
mentioned above to enhance teaching/leaming culture to the advantage of the institution, faculty and 
students. One part of the management framework is the use of the 3AF feedback form. The 3AF feedback 
form has only 6 necessary ratings that take less than 5 minutes to complete. Hence, the form can conveniently 
be used many times by the lecturer during the course for in-course tracking of teaching quality. At the end 
of the course it can be used by the administration to give a single decision point number representing the 
quality of teaching. This is also fairer and less threatening to faculty who, by previous uses of the form, 
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have had opportunities to respond to the feedback and so improve the course for their students and for 
themselves. As mentioned above, previous research has connected teaching and assessment problems on 
professional courses with staff/student mis-matched expectations of three abilities. These three abilities 
are technical skills, professional competence and professional attitudes (Bastick, 1995). The 3AF uses the 
alignment of staff/student expectations on these three abilities as the indicator of teaching effectiveness. 

The 3AF form asks for two ratings of each of these abilities; ratings of how it is now on the course and 
ratings of how the student would want it to be. This is shown in figure 1 . The form takes longer to complete 
when additional information is requested, such as age, sex, etc., used for targeting results to student 
subgroups. 



Estimate, for vou personally, how much this course emphasises, and should emphasise (i) Skills, 
(ii) Competence and (iii) Attitudes? Do this for both how the course is now, and for how the 
course should be - write a number in each box. 

As it is now on this course As it should be on this course 



(i) Emphasis on Skills 



Your 
estimate 
out of 100 



Your 
estimate 
out of 1 00 



As it is now on this course As it should be on this course 



(ii) Emphasis on Competence 




Your 
estimate 
out of 100 



As it is now on this course As it should be on this course 



(iii) Emphasis on Attitudes 



Your 
estimate 
out of 100 



Your 
estimate 
out of 1 00 



Figure 1: 3AF feedback form asking for two ratings of each of the three abilities 

Technical Skills refer to the traditional speed and accuracy of reproducing facts and processes and is 
assessed by timed accuracy of reproduction. Professional competence refers to the ability to use the skills 
in a novel situation or extend these skills in a novel way. The assessment is by justification of the 
appropriateness of what is done. Professional attitudes refers to values that are appropriate to the subject. 
They are assessed by demonstration in practical situations. 

From the difference in each pair of ratings it is possible to calculate each student’s expectation for 
change in that ability. The lecturer completes the same ratings at the same time, and from the lecturer’s 
form it is possible to also calculate, in the same way, the lecturer’s expectations for change in each of the 
three abilities. Previous research has shown that when the students’ expectations are the same as the 
lecturer’s, that is both students and lecturer are working towards the same degree of change (indicator of 
effectiveness), then students get high grades and both the lecturer and the students enjoy the course (criteria 
of effectiveness). The correlations between the in-course indicators and the post-course criteria are 
significant at p<0.01 (Bastick, 1995). 

Measuring Quality of Teaching (QT) 

The 3AF involves more than using the feedback form and calculating the results. There are four steps 
in complete the application of the method. In steps 1 and 2 faculty are supported as part of their institutional 
staff and course development programme which widens their awareness of quality teaching techniques. 
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1. Pre-course peer justification of ratings 

In the design stage, before the course starts, the lecturer needs to use his/her professional expertise to 
decide on what should be the emphasis on the course for each of the three abilities. A rationale should 
justify this decision, and it needs to be peer agreed. The lecturer can then build his/her expectations of the 
three abilities into the teaching and the design of course assignments that give students the opportunities to 
demonstrate the required level of each ability. 

2. Explain and justify three abilities to students 

Soon after the beginning of the course the lecturer needs to make sure the students understand the 
three abilities and how they will be taught and assessed through the content of the course. 

3. Monitor ‘as-is* and ‘should-be ’for students and lecturer 

The 3 AF feedback form is completed by students and the lecturer when the lecturer is ready to monitor 
the course, or the administration is ready to assess the teaching. 

4. Calculating the quality of teaching 

This is done in the following two stages (a) and (b). 

a) Calculate expected change for each student and lecturer. 

Change=| ‘ should-be’ -‘as-is ’ |/‘as-is ’ 

b) Calculate alignment: 

Alignment=|Lecturer change- Student change| 

Zero is the perfect score 

The alignment is the Quality Teaching (QT) score. The QT score can be calculated for each student 
or as a mean for any group of students. Hence, the quality of teaching can be monitored for any taiget group 
of interest e.g. older students, students taking special electives, minority groups, gender balance, etc. 
Adjustments can be made to in-course teaching as necessary. 

Safeguards from variation in students’ set expectations 

Students’ expectations vary across Faculties and subjects, by years of education, previous experience 
and even by whether the course is compulsory or optional (Goldman, 1993). This creates an ‘unlevel 
playing field’ when traditional opinionnaires are used to assess faculty teaching. From the 3AF feedback 
form the variation in students’ expectations can be calculated and the result used as a safeguard to protect 
the lecturer from inappropriate student expectations. It will be realized that all course evaluations depend 
on matching student expectations to the expectations of the lecturer for course. The quality of teaching is 
a function of the complex relationship between the student, the course and the lecturer. All three contribute 
to the responsibility for quality teaching as assessed by the alignment of expectations. Some aspects of the 
course can be presented in different ways to match different student expectations. For example, so called 
‘learning styles’ can be matched by adopting different ‘teaching styles’. However, some aspects of the 
course may not be open to change to match student expectations, e.g. externally accredited content standards 
or the peer agreed emphases of the three abilities. Just as students expectations vary so does the flexibility 
of their expectations. If students’ expectations of these unchangeables cannot be altered so they can accept 
them, then teaching ratings will go down through no fault of the lecturer. Traditional opinionnaires penalize 
the lecturer because they make no allowance for large variations in unchangeable student expectations. 
However, the 3AF allows the lecturer to show evidence that the original peer agreed emphases may not be 
appropriate for some groups on the course. This evidence can be used to either change the course expectations 
or change the student selection criteria. 
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Institutional commitment to staff development 

The full Framework includes the commitment of the institution to develop faculty’s ability to use 
their subject specialism as a vehicle for explaining, teaching and assessing the three abilities. Institutional 
staff development support includes promoting academic freedom and professional responsibility, assisting 
faculty in making expectations explicit, in designing assessment opportunities for the three abilities and 
developing the ability of faculty to teach the three abilities using the content of their subject areas. As was 
mentioned at the start of this article, there is a saying in business that “what gets measured gets done and 
what gets rewarded gets repeated” (Friend, 1972). Quality teaching and quality learning get measured and 
get rewarded by the 3AF. 

Summary of the 3AF and how it promotes a positive teaching and learning culture 

To use the 3AF in practice, lecturers explain to their students the three abilities that underlie quality 
teaching and learning and explain how they will be taught and assessed. When they wish to monitor the 
effectiveness of their teaching they ask the students to rate how they see the current emphasis of these three 
abilities and to rate how they would prefer the emphasis to be. The lecturer makes the same rating of the 
course. The indicator of effective teaching is that the students and the lecturer are working towards the 
same changes in these three abilities. This is measured by ‘the change expected by the students’ subtracted 
from ‘the change expected by the lecturer’. Zero is the perfect score on the total of the three abilities, and 
indicates perfect alignment. The alignment score is the measure of effective teaching and can be calculated 
for individual students, and the mean calculated for minority groups or for special comparisons e.g. to 
measure if the teaching more effective for male than for female continuing education students. 

The method promotes a positive teaching and learning culture both directly and indirectly. It promotes 
a positive teaching and learning culture indirectly by encouraging fonns of teaching and learning that 
faculty and students use to increase their valued assessment results, i.e. assessment driven teaching and 
learning. Namely, this method encourages teaching and learning that promotes students’ critical and 
evaluative thinking, high standards in technical skills and professional values because this is what is assessed 
in faculty teaching and in students learning. 

The assessment method also promotes a positive teaching and learning culture directly through student 
and faculty assessment support processes, as follows: 

1 . The institution promotes academic freedom and professional responsibility by confirming the 
lecturers’ professional prerogative to decide, and justify to their peers and their students, the emphasis 
they judge should be given to each of the three abilities on their courses. This is reinforced by 
recognising an assessment process that lecturers control. 

2. The institution promotes professional development by assisting faculty in making their professional 
expectations explicit in terms of the three abilities in their subject area and in assisting them to model 
them in their teaching and to design coursework and examinations that offer opportunities for assessing 
these three abilities in their subject areas. 

3. Faculty encourage students’ critical and evaluative thinking, to the extent faculty can justify this as 
desirable, their quality teaching and by not assessing the correctness of professional competence, but 
by assessing the students’ justifications of why the novel aspects of their applications are appropriate. 

4. Faculty also explicitly encourage professional attitudes, to the extent they can justify these as 
desirable, by their teaching and by assessing demonstrations of professional attitudes on course 
assignments. 
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Generally, the development of technical skills is already well served by traditional methods of 
assessment. Now staff development programmes need to be brought on-line to share successful methods 
of teaching and assessing professional competence and professional values. Processes 3 and 4 above directly 
promote the research culture of the institution by raising faculty and student awareness of the importance 
of critical and evaluative thinking, rewarding faculty for enhancing these abilities in their students and 
explicitly rewarding faculty and students for demonstrating these abilities. 

Current compartmentalised use of anonymous student opinionnaires for the assessment of teaching 
has been shown to undermine teaching/leaming culture. By integrating teaching, learning and assessment 
with curriculum, research and resourcing aspects of quality assurance in the ways outlined in this paper, 
the 3 AF empowers staff to continuously improve the teaching and learning culture of their institution. For, 
as argued by Harvey and Knight (1996), ‘the key to quality improvement lies in empowering academic 
staff to undertake a process of continuous quality improvement in relation to student learning’. More 
appropriately prepared students will then feed into higher degree programmes and find more appreciative 
employers in the job markets. These increased competency outcomes can only further demonstrate the 
success of the institution as a continuing education provider. 
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